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Summary of response to the consultation on ICO 
guidance on the AI auditing framework, with 
comments 


Introduction 


In February 2020, the IC® published draft guidance en the AI auditing 
framewerk, with an initial deadline ef 1 April 2020 fer cemments. Bue te the 
cerenavirus pandemic, this deadline was extended until 1 May 2020. 


@ur survey asked fer: 


e feedback en hew well pitched each sectien ef the guidance was; 

e views en the list ef centrels erganisatiens ceuld use te mitigate seme 
ef the risks Al peses te individual rights; 

e practical examples that ceuld further help eur thinking, and 

e previded an eppertunity fer respendents te make any further general 
cemments. 


The IC@ weuld like te thank all these erganisatiens and individuals whe teek 
the time te read the draft guidance and give us their views, and these whe 
effered te werk with us further. We are espeq@ally grateful that even in times 
ef a glebal pandemic, yeu made time te engage with us en this guidance. We 
have carefully neted all yeur cemments, and these have been invaluable in 
shaping eur thinking en this tepic as we preduced the final versien ef the 
guidance, 


Quantitative summary 


@verall, we received eSrespenses te eur draft guidance. 48 respendents 
fellewed the survey questiens, and 17 sent a standalene decument with their 
feedback te us directly. 


Bue te the way the survey was run, the largest prepertien ef respenses 
received was frem unknewn secters (30). @f the seurces knewn, the private 
secter had the largest prepertien. The distrisutien ef respenses is shewn in 
Figure 1 belew. 
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Sectors represented by respondents 
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Figure 1: Secters represented by these that respended te the censustatien 
@verall, the respense te eur censultatien was generally pesitive. 
Is the draft guidance clear about what you should consider 


when creating and using Al systems that are compliant with 
data protection law? 


Yes 


Figure 2: respenses te Whether the draft guidance wes clear 


Is it easy to find information in the draft guidance? 
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Figure 3: respendents en whether it was easy te find infermatien in the draft guidance 


Ne respense 
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Are the risk statements and the examples of controls 
useful? 


Yes Ne 


Figure 4: respendents en whether the nsk statements and examples ef centrefs were useful 


Specific sectiens received a mere mixed respense. The quantitative 
respenses te the sectiens are illustrated belew. 
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Figure 5: respenses te hew well pitched was the ‘Abeut this guidance’ sectien 


What are the accountability and governance implications of 
Al? 


20 
15 
10 
E 
n emr, 


Jus: right Tee vague Tee detailed Ne respense 


Figure é: respenses te hew well pitched was the sectien en acceuntabilitY and gevemance 
imptlicatiens ef AI 
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What do we need to do to ensure lawfulness, fairness, and 
transparency in Al systems?’ 


20 
15 
10 
E 
8 es 


Jus: right Tee vague Tee detailed Ne respense 


Figure 7: respenses te hew well pitched was the sectien en lawfulness, faimess, and transparency 
in AI systems 


How should we assess security and data minimisation in Al? 
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Figure 8: respendents en hew well pitched the sectien en security and data minimisatien in AI was 


How do we enable individual rights in our Al systems? 
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Figure 9: respenses te hew well pitched the sectien en individual nght and AI systems was 


While we cannet respend individually te each centributien, we have previded 
an everview belew ef the key themes that have beceme apparent and seme 
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cemments en eur emerging thinking frem each area ef the censultatien as 
we finalised eur guidance. 


Key themes 
About this guidance 


In the draft guidance, we stated that the AI auditing framewerk will previde 
teels and reseurces te assist eur investigatien and assurance teams when 
assessing the cempliance ef erganisatiens using Al. Seme respendents 
queried whether we have the pewer te ‘audit algerithms’ themselves. 


Seme respendents were unsure what the status ef the guidance was. They 
did net knew whether it was guidance en best practice, er guidance en 
interpreting data pretectien law (er beth). There was additienal cenfusien 
abeut the guidance being pertrayed as best practice but alse being used as a 
standard fer eur investigatien and assurance teams te assess eempliance. 


Seme respendents criticised eur definitien ef ‘artificial intelligence’. Seme felt 
it was tee bread and included systems that weuld net erdinarily be referred 
te as Al. 


Finally, there were respenses that suggested that mere ceuld be dene te 
signpest which sectiens weuld require assistance frem technelegy experts te 
interpret what we say. 


@ur respense 


The Data Pretectien Act 2018 (BPA 18) gives us the pewer te carry eut 
varieus auditing and investigatien activities. We believe that, in seme cases, 
this includes the recevery and analysis ef evidence, including the AI systems 
themselves. 


The guidance inferms yeu abeut what we think censtitutes best practice fer 
data pretectiencemplaint AI as well as hew we interpret data pretectien law 
as it applies te AI systems that precess persenal data. We take en beard that 
this ceuld have been made clearer and have taken steps te remedy this in 
the guidance. Fer example, we have advised that where we use ‘sheuld’, yeu 
sheuld interpret this as best practice, but where we use ‘must’, yeu sheuld 
interpret this as a legal ebligatien under data pretectien law. 


We acknewledge that eur initial definitien ef Al ceuld be interpreted breadly. 
Hewever, neither the General Bata Pretectien Regulatien (GBPR) ner the BPA 
18 defines ‘artificial intelligence’, se nene ef yeur legal ebligatiens depend en 
exactly hew itis defined. We think it is useful te understand breadly what we 
mean by Al in the centext ef the guidance. We have included twe definitiens 
in the guidance: ene used within the Al research cemmunity and ene used in 
the data pretectien centext. We use the umbrella term ‘Al’ because it has 
beceme a standard industry term fer a range ef technelegies, including 
machine learning. 
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We take en beard that we ceuld have been clearer te signpest sectiens which 
weuld likely need assistance frem a technelegy expert te interpret what we 
mean and have taken steps te cerrect this. Fer example, we have included a 
Subsectien at the start ef each part where we describe whe this is fer. We 
have alse added a glessary ef seme ef the technical terms used fer 
reference. 


What are the accountability and governance implications of AI? 


There were several cemments en eur guidance abeut when te carry eut a 
data pretectien impact assessment (BPIA). Respenses indicated that it is net 
a legal requirement fer erganisatiens te carry eut a BPIA when Al is being 
used te precess persenal data and that the draft guidance misinterpreted the 
law when it said it is. 


Semerespendents suggested that seme activities we interpreted as typically 
being carried eut by centrellers ceuld be interpreted as precesser activities. 


We received several requests frem respendents te include mere ‘trade-effs’ 
that individuals in cempliance fecus reles are likely te face. Seme felt that we 
had underplayed the risks asseciated with cemmercial sensitivity when 
discussing the trade-eff, it may have with explainability. 


Finally, the mathematical selutiens te assessing trade-effs that were included 
in the draft guidance was questiened by several respendents abeut hew 
practical they weuld be in a real-werd envirenment. 


Our response 


We take en beard the peint that net all applicatiens ef Al that precesses 
persenal data is likely te result in high risk and therefere trigger the legal 
requirement te undertake a DPIA. Hewever, we think that in the vast 
majerity ef Al systems that precess persenal data, a BPIA will need te be 
undertaken. In any case, it is geed practice te carry eut a BPIA fer a new 
preject even if yeu are net legally required te dese. We recegnise that yeur 
assessment ef whether a BPIA needs te be undertaken will be made ena 
case by case bases. In these cases where yeur assessment reaches the 
cenclusien that an AI system is net likely te result in high risk precessing, 
yeu will still need te decument hew yeu came te that decisien. 


We have alse mere clesely aligned this sectien with eur existing GB PR 
guidance en determining whether yeu are a centreller er a precesser te 
ensure eur appreach remains censistent. 


@ur werk has identified that, when AI systems invelve a number ef 
erganisatiens in the precessing ef persenal data, assigning the reles ef 
centreller and precesser can beceme cemplex—fer instance, when seme ef 
the precessing happens in the cleud. This raises questiens ef pelicy, and we 
plan te werk with Gevernment te explere these areas, with a view te 
addressing these issues in mere detail when we revise eur Cleud Cemputing 
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Guidance in 2021. This Guidance will alse be subject te external stakehelder 
censultatien prier te its finalisatien. 


We understand that there are likely te be several cempeting interests that 
yeu will have te censider when designing and develeping an AI system. @®ur 
thinking has develeped en this and we new think that ‘trade-effs’ is a 
misleading term when talking abeut these cempeting interests in a data 
pretectien centext, as it ceuld imply that yeu can trade away ene legal 
ebligatien fer anether when this is net the case. Instead, we think that it is 
abeut striking the right balance between cempeting interests while ensuring 
yeu cemply with yeur ebligatien under data pretectien law. In the latest 
guidance, we have reframed ‘trade-effs’ as ‘cempeting interests’ when 
talking abeut them in a data pretectien centext. In additien, we have 
remeved the sectiens which discuss specific cempeting interests and have, 
where apprepriate, embedded them in the relevant sectiens ef the guidance. 


We agree that the mathematical selutiens that we described in the draft 
guidance were net as helpful as we initially theught. Therefere, we have 
decided te remeve the werked example frem the latest guidance te aveid 
cenfusien, We have kept a shert discussien en mathematical appreaches but 
nete that they can be difficult te meaningfully put them inte practice, 


What do we need to do to ensure lawfulness, fairness, and 
transparency in AI systems? 


We were asked te previde an example ef an unfair Al system when 
discussing the fairness principle. 


Seme respendents requested fer further detail en when legal ebligatien is 
likely te be an apprepriate lawful basis fer precessing persenal data in the 
centext ef an AI system. 


We received a suggestien te talk abeut the apprepriateness ef the legitimate 
interests lawful basis in the centext ef the initial research and develepment 
phase ef an AI system, where purpeses may be quite bread. 


There was seme useful censtructive criticism abeut eur discussien ef the 
fairness principle and where it everlaps with the UK’s anti-diseri minatien legal 
framewerk, netably the UK Equality Act 2010 (EA2010). It was neted by 
seme that what we say erganisatiens must er sheuld de te cemply with the 
fairness principle weuld net necessarily mean that they have cemplied with 
their ebligatiens under the EA2@1@. It was peinted eut te us that net all 
disparities in a data pretectien centext weuld be instances ef discriminatien 
under equalities law. We were made aware that net all cases ef 
discriminatien will censtitute unlawful discriminatien under equalities law. 


Semerespendents previded further causes ef petentially discriminatery Al 
systems that were net discussed in the draft guidance. 
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We were made aware ef the cenflict between algerithmic fairness and 
relevant nen-discriminatien law in the UK. Fer example, seme ef the 
techniques we discussed in the centext ef algerithmic fairness weuld net 
mitigate the risks ef nen-cempliance with nen-discriminatien law. 


Several respendents mentiened cempeting interests when assessing 
discriminatien in an Al system. Fer example, cellecting mere data ena 
minerity pepulatien teimpreve the statistical accuracy ef the Al system and 
the risk ef nen-cempliance with the data minimisatien principle. Anether 
example previded was where yeu may have ether secter-spe@fic regulatery 
ebligatiens regarding statistical accuracy er discriminatien which need te be 
censidered alengside yeur data pretectien ebligatiens. 


Finally, semerespendents wanted greater clarificatien abeut when te identify 
a lawful basis under Article @ er 1@ and Article @ in cases where an AI system 
may infer special categery data er criminal cenvictiens data (er beth). 


Our response 


We acknewledge the request fer an example ef an unfair Al system and have 
included seme in the guidance. 


We recegnise that that yeur erganisatien may be required te audit yeur Al 
systems te ensure they are cempliant with varieus legislatien (including but 
net limited te data pretectien), and this may invelve precessing ef persenal 
data, fer instance te test hew the system perferms en different kinds ef 
peeple. Such precessing ceuld rely en legal ebligatien as a basis, but this 
weuldenly cever the auditing and testing ef the system, net any ether use ef 
that data. Yeu must be able te identify the ebligatien in questien, either by 
reference te the specific legal previsien er else by peinting te an apprepriate 
seurce ef advice er guidance that sets it eut clearly. We believe, theugh, that 
itis unlikely that it will be necessary te use Al te carry eut this ebligatien. 


We take en beard the suggestien abeut legitimate interests as a lawful basis 
where initial research and develepmentis taking place with bread purpeses. 
We believe that legitimate interests may be an apprepriate lawful basis 
depending en the circumstances, and recemmend that, in seme cases, as 
mere specific purpeses are identified, yeu review yeur legitimate interests 
assessment accerdingly (er identify a different lawful basis). 


We appreciate the cemments and suggestiens we received abeut the UK’s 
anti-discriminatien legal framewerk and hew it relates te the fairness 
principle in data pretectien law. We have attempted te make it clearer that 
the guidance is enly directed at hew te cemply with the fairness principle and 
what best practice leeks like. Where it has relevance te the UK’s anti- 
discriminatien legal framewerk, we have neted it. This includes where what 
we say weuld net necessarily mean yeu will cemply with anti-discriminatien 
law. We have alse made it clearer what eur interpretaten ef the fairness 
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principle is. This will make clearer where the principle will everlap with anti- 
discriminatien law and where it will diverge. 


We alse appreciate the suggestiens ef further causes ef petentially 
discriminatery Al systems and have included seme in the guidance. 


We agree with the peints made abeut techniques used te ensure algerithmic 
fairness and the petential cenflict with equalities law. We have clarified in the 
guidance that these techniques may net be suitable te cemply with equalities 
law, 


We nete the pessible cempeting interests when addressing diseriminatien in 
an Al system and have included seme werds en seme ef these in the 
guidance. 


We recegnise that it may be difficult te identify whether an Al system has 
accidentally inferred spe@al categery data er criminal effence data (er beth). 
If itis unclear whether er net yeur system may be inferring such data, yeu 
may want te identify a cenditien te cever that pessibility and reduce yeur 
cempliance risk, altheugh this is net a legal requirement. Whether there 
intent te infer special categery data er whether there is a reasenable degree 
ef certainty that yeu haven't inferred special categery data (er beth) is alse 
relevant. 


How should we assess security and data minimisation in Al? 


Several respendents cemmented that they did net think the guidance en 
security and Al systems was specific eneugh te Al. They suggested mere 
discussien abeut specific security risks that Al creates and exacerbates. 


Seme respendents suggested that erganisatiens using AI te precess persenal 
data sheuld take a mere helistic appreach when it cemes te assessing the 
security ef Al systems. They highlighted that Al systems are just ene 
cempenent ef alarger chain ef seftware cempenents, data flews, 
erganisatienal werkflews and business precesses, 


We were advised te discuss everfitting as being a pessible reasen yeu might 
cite te justify having mere data peints. 


Several respenses suggested that itis semetimes difficult te guarantee that 
ne persenal data is inadvertently shared. @ur draft guidance stated that yeu 
remain respensible fer ensuring persenal data is net expesed. It was felt by 
seme that this was an unreasenable expectatien ef centrellers. 


Respendents peinted eut that mathematical metheds such as differential 
privacy are net sufficiently mature eneugh te depley in a real-life centext. 


It was suggested te us that the guidance includes infermatien abeut 
synthetic data and hew it ceuld be used te help cempliance with the data 
pretection principle. 
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Finally, we received seme respenses that suggested that ‘faceprints’, in the 
centext ef facial recegnitien medels, are net persenal data because they are 
enly identifiable te a specific medel. 


Our response 


We appreciate that the sectien en security ceuld have been mere specific te 
Al. We have remeved seme mere general parts. Hewever, we believe that, 
everall, this sectien was applicable eneugh te Al even if it was alse applicable 
te nen-Al systems. 


We agree that a helistic appreach te security is werthwhile and effective. We 
have included a line suggesting that erganisatiens fellew this appreach. 


We acknewledge the peint abeut everfitting. We have suggested that 
everfitting can happen where there are tee many features included er where 
there are tee few examples in the training data er beth. 


We agree with the peint made abeut the difficulties ef ensuring that persenal 
data used te train yeur medels is net expesed because ef the way yeur 
clients have depleyed the medel. We have rephrased eur expectations here 
te yeu being respensible fer assessing and mitigating the risk that persenal 
data used te train yeur medels may be expesed by yeur clients depleying the 
medel. By deing this, we remeve the expectatien ef yeu mitigating the risk 
cempletely and instead ept fer a mere prepertienate risk-based appreach. 


We have caveated the discussien abeut differential privacy, stating that it 
may net be apprepriate er sufficiently mature te depley in yeur particular 
centext. We will centinue te meniter develepments and update the guidance 
accerdingly. 


We have included seme infermatien abeut synthetic data and hew it ceuld be 
used, in seme cases, te help yeu cemply with the data minimisatien principle. 
Altheugh, we nete that there are risks assecated with synthetic data being 
de-identified and the data net being useful fer yeur purpeses. Further 
guidance en synthetic data will be published in eur anenymisatien werk. 


We disagree that ‘faceprints’, in the centext ef facial recegnitien medels, are 
net persenal data. We think they are very much identifiable within the 
centext ef the specific facial recegnitien medels that they are created fer. 
When used fer the purpeses ef uniquely identifying an individual, they weuld 
be special categery data under data pretectien law. 


How do we enable individual rights in our AI systems? 


We received seme feedback that suggested that it may be impessible fer 
centrellers te facilitate individual rights because their eutseurced services de 
net get the infermatien er functienality they need. 


Several respendents questiened why we had net included guidance en 
specific individual rights. 
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We were asked fer seme clarificatien abeut whether individuals needed te be 
infermed if their data was geing te be precessed te train an AI system if the 
centreller did net knew that they were geing te use if fer this purpese when 

they first cellected it. 


Seme respendents disagreed that retraining and redepleying a medel sheuld 
net be prehibitively cestly. This was in the centext ef fulfilling requests like 
erasure er rectificatien in medels that centain persenal data by design. 


We were asked whether individuals have the right te meaningful infermatien 
abeut the legic invelved in an Al-driven decisien where these decisiens are 
net selely autemated er where they denet preduce legal er similarly 
significant effects er beth. 


Our response 


We recegnise that if yeu eutseurce an Al service te anether erganisatien, this 
ceuld make the precess ef respending te rights requests mere cemplicated 
when the persenal data invelved is precessed by them rather than yeu. When 
precuring an AI service, yeu must cheese ene which enables individual rights 
te be pretected te meet yeur ebligatiens as a centreller. If yeur chesen 
service is net designed te easily facilitate such rights, this dees net remeve 
er etherwise change these ebligatiens. If yeu are eperating as a centreller, 
yeur centract with the precesser must stipulate that the precesser assist yeu 
in respending te rights requests. If yeu are eperating an Al service as a jeint 
centreller, yeu need te decide with yeur fellew centreller(s) whe will carry 
eut which ebligatiens. 


We acknewledge that we did net discuss all individual rights. These rights are 
still impertant, and yeu must enable them (where applicable), Hewever, we 
decided te enly include rights where AI creates er exacerbates the risk ef net 
enabling them. Fer guidance en individual rights net included in this 
guidance, read eur Guide te the GBPR. 


Yeu must inferm individuals if their persenal data is geing te be used fer the 
purpeses ef training an Al system, te ensure that precessing is fair and 
transparent. This infermatien sheuld be previded at the peint ef cellectien. If 
the data were initially precessed fer a different purpese, and yeu later decide 
te use it fer the separate purpese ef training an Al system, yeu will need te 
inferm the individuals cencerned (as well as ensuring the new purpese is 
cempatible with the previeus ene). 


We acknewledge that in seme cases it ceuld be prehibitively cestly te retrain 
and redepley a medel that centains data by design. We believe that it will be 
less cestly te retrain and redepley yeur Al medels accerdingly if yeu have a 
well-erganised medel management system and depleyment pipeline. 
Hewever, we recegnise that this is may net be a legal requirement. 
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We have clarified that individuals enly have the right te meaningful 
infermatien abeut the legic invelved in an Al-driven decisien where that 
decisien is selely autemated and has legal er significantly similar effects. 
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